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Abstract. Symmetry of information establishes a relation between the 
information that x has about y (denoted I(x : y)) and the information 
that y has about x (denoted I(y : x)). In classical information theory, the 
two are exactly equal, but in algorithmical information theory, there is 
a small excess quantity of information that differentiates the two terms, 
caused by the necessity of packaging information in a way that makes 
it accessible to algorithms. It was shown in |Zimll| that in the case 
of strings with simple complexity (that is the Kolmogorov complexity 
of their Kolmogorov complexity is small), the relevant information can 
be packed in a very economical way, which leads to a tighter relation 
between I(x : y) and I(y : x) than the one provided in the classical 
symmetry-of-information theorem of Kolmogorov and Levin. We give 
here a simpler proof of this result. This result implies a van Lambalgen- 
type theorem for finite strings and plain complexity: If x is c-random 
and y is c-random relative to x, then xy is 0(c)-random. We show that a 
similar result holds for prefix-free complexity and weak-K-randomness. 

1 Introduction 

In classical information theory, the information that X has about Y is equal to 
the information that Y has about X, i.e., 

I(X :Y)=I(Y :X)B 

In algorithmical information theory, there exists a similar relation, but it is 
less perfect: 

I(x : y) « I(y : x). 

In this paper we take a closer look at the level of approximation indicated by "sa." 
In the line above, x and y are binary strings of lengths n x and, respectively, n yi 
and I(x : y), the information in x about y, is defined by I(x : y) = C(y | n y ) — 
C(y | x, riy), where C(-) is the plain Kolmogorov complexity. The precise form of 
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1 X and Y are random variables and I(X : Y) = H(Y) - H(Y | X), where H is 
Shannon entropy. 
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the equation is given by the classical Kolmogorov-Levin theorem |ZL70j , which 
states that 

\I(x : y) - I(y : x)\ = 0{\ogn x + \ogn y ). (1) 

This is certainly an important result, but in some situations, it is not saying 
much. Suppose, for example, that x has very little information about y, say, 
I(x : y) is a constant (not depending on x and y). What does this tell about 
I(y : a;)? Just from the definition and ignoring I{x : y), I(y : x) can be as large as 
n x — O(l), and, somewhat surprisingly, given that I(x : y) = 0(1), I(y : x) can 
still have the same order of magnitude. As an example, consider y a c-random 
string (meaning C(y \ n y ) > n y — c), whose length n y is also c-random. Let x be 
the binary encoding of n y . Then I{x : y) — C(y \ n y ) — C(y | x, n y ) = 0(1), but 
I(y : x) = C{x | n x ) — C(x \ y, n x ) = n x — O(l). This example shows that even 
though, for some strings, the relation (TTJ) is trivial, it is also tight. 

In many applications, the excess term 0(\ogn x + logfij,) does not hurt too 
much. But sometimes it does. For example, it is the reason why the Kolmogorov- 
Levin Theorem does not answer the following basic "direct product" question: 
If x and y are c-random n-bit strings and I(x : y) < c, does it follow that xy is 
0(c)-random? 

The answer is positive (providing the finite analog of the van Lambalgen the- 
orem) . It follows from a recent result of the author |Zimllj , which establishes a 
more precise symmetry-of-information relation for strings with simple complex- 
ity, i.e., for strings x such that C^(x \ n x ) is small, say, bounded by a constant^] 
Note that all random strings have simple complexity, and that there are other 
types of strings with simple complexity as well. The main result in [Zimll] im- 
plies that if x and y are strings of equal length that have simple complexity 
bounded by c and I(x : y) < c, then I(y : x) < 0(c). 

The proof method in [Zimll j is based on randomness extraction. Alexander 
Shen (personal communications) has observed that in fact a proof of the above 
result in Zi mll] can be obtained via the standard method used in the proof of 
Kolmogorov-Levin Theorem (see }DH10|LV08j ). We present here such a proof. 

We slightly extend the symmetry-of-information relation from [Zimll] to 
strings x and y that may have different lengths. We prove the following (with 
the convention that logO = 0): 

Theorem 1 (Main Theorem). For all strings x and y, 

I(y : x) < I(x : y) + 0(\ogI(x : y)) + 0(\og | I) +S(x,y), 

where S(x,y) = 0(C^(x | n x ) + C™(y \ n y )). 

Thus, for strings x and y with simple complexity, I{y : x) < I(x : y) + 
0(log/(a; : y)) + 0(log \n x - n y \). 

As mentioned, Theorem Q] implies an analog of the van Lambalgen Theorem 
for finite strings and plain Kolmogorov complexity. Recall that van Lambalgen 

2 The notation C^ 2 ' (x \ n x ) is a shorthand for C(C(x \ n x ) \ n x ). 
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Theorem |vL87) states that for infinite sequences x and y the following two 
statements are equivalent: 

(1) x is random and y is random relative by x, 

(2) x (By is random. 

(Here, random means Martin-L6f random, and x © y is the infinite sequence 
obtained by alternating the bits of x and y). Theorem [T] implies immediately the 
following. 

Theorem 2 ([Zimll]). Let x and y be two strings of length n such that x is 
c-random and y is c-random relative to x. Then xy is 0(c)-random. 

Formally, if C(x n) > n — c, C (y | x) > n — c, then C(xy \ In) > 2n — O(c). 

(The constant in the O(-) notation depends only on the universal Turing 
machine.) 

Is there a van Lambalgen Theorem analog for prefix-free complexity? To 
adress this question, we note that there are two notions of randomness for 
the prefix-free complexity K, weak i^-randomness, and strong X-randomness 
(see |DH10j ). An n-bit string x is 

(1) weakly c-(K-random) if K (x | n) > n — c, and 

(2) strongly c-(K-random) if K{x \ n) > n + K(n) — c. 

For weak X-randomness, we prove in Section [4] the following analog of the 
van Lambalgen Theorem. For strong fC-randomness, the question remains open. 

Theorem 3. Let x and y be two strings of length n such that x is weakly c- 
(K-random) and y is weakly c-(K-random) relative to x. Then xy is weakly 
0(c)-( K-random). 

Formally, if K{x \ n) > n — c,K(y \ x) > n — c, then K(xy \ 2n) > In — 0(c). 
(The constant in the O(-) notation depends only on the universal Turing 
machine.) 

1.1 Proof techniques 

The proofs of Symmetry of Information Theorems have a combinatorial nature. 
To fix some ideas, let us first sketch the proof of the classical Kolmogorov- 
Levin Theorem which establishes relation (TTJ). The setting of the theorem, as 
well as for the rest of our discussion, is as follows: x and y are binary strings of 
lengths n x and, respectively, n y . An easy observation (see Section [2~2l) . shows 
that a relation between L(y : x) and L(x : y) can be deduced from a "chain 
rule:" C(xy \ n x ,n y ) > C(x \ n x ) + C(y \ x,n y ) — (small term). In our case, to 
obtain <JTJ , it is enough to show that 

C(xy | n x , n y ) > C(x \ n x ) + C(y \ x,n y ) - 0{\ogn x + \ogn y ). (2) 

Let us consider a 2™ x x 2 Uy table with rows indexed by u £ {0, 1}™ X and columns 
indexed by v £ {0, 1}™». Let t = C(xy \ n x , n y ). We assign boolean values to the 
cells of the table as follows. The (u, u)-cell in the table is 1 if C{uv \ n Xl n y ) < t 
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and it is otherwise. The number of cells equal to 1 is less than 2' +1 , because 
there are only 2* +1 programs of length < t. Let m be such that the number of l's 
in the x row is in the interval (2 m_1 , 2™ 1 ]. Note that, given x, n y and t, we can 
enumerate the 1-cells in the x-row and one of them is the (x, y) cell. It follows 
that 

C(y | x,n y ) < m + OQogt). (3) 

Now consider the set of rows that have at least 2 m ~ 1 Is. The number of such 
rows is at most 2 t+1 /2 m ~ 1 = 2'~ m+2 . We can effectively enumerate these rows 
if we are given n Xl n yi m and t. Since the x row is one of these rows, it follows 
that 

C(x | n x ) <t — m + Oi\ogn y + logm + \ogt). (4) 

Adding equations ([3]) and (fjij and keeping into account that log m < log n y and 
\ogt < 0(logn x + \ogn y ), the relation ([2]) follows. 

A careful inspection of the proof reveals that the excess term 0(logn x + 
logn y ) is caused by the fact that the enumerations involved in describing y and 
x need to know m (which is bounded by C(y | n y )) and t = C(xy | n x ,n y ). In 
case x and y are c-random, it is more economical to use randomness deficiencies. 
In particular instead of using t — C(xy \ n x ,n y ), we can do a similar argument 
based on the randomness deficiency of xy, i.e., w = (n x + n y ) — C{xy \ n x , n y ). 
This is an observation of Chang, Lyuu, Ti and Shen [CLTS09] , who attribute it 
to folklore. For strings with simple complexity, it is advantageous to express t as 
t = C(x \ n x ) + C(y | x) — w because the first two terms have a short description. 
This yields a proof of the Main Theorem which we present in Section [3] 

2 Preliminaries 

2.1 Notation and background on Kolmogorov complexity 

The Kolmogorov complexity of a string x is the length of the shortest effective 
description of x. There are several versions of this notion. We use here mainly the 
plain complexity, denoted C(x), and the conditional plain complexity of a string 
x given a string y, denoted C(x | y), which is the length of the shortest effective 
description of x given y. The formal definitions are as follows. We work over 
the binary alphabet {0, 1}. A string is an element of {0, 1}*. If x is a string, n x 
denotes its length. Let M be a Turing machine that takes two input strings and 
outputs one string. For any strings x and y, define the Kolmogorov complexity of 
x conditioned by y with respect to M, as Cm{x \ y) — min{|p| | M(jp,y) = x}. 
There is a universal Turing machine U with the following property: For every 
machine M there is a constant cm such that for all x, C\j(x \ y) < Cm(x \ y) + 
cm- We fix such a universal machine U and dropping the subscript, we write 
C(x | y) instead of Cu(x | y). We also write C(x) instead of C(x | A) (where 
A is the empty string). If n is a natural number, C(n) denotes the Kolmogorov 
complexity of the binary representation of n. We use C^ 2 \x\n x ) as a shorthand 
for C(C(x | n x ) | n x ). For two strings x and y, the information in x about y is 
denoted I(x : y) and is defined as I(x : y) — C(y | n y ) — C(y | x,n y ). 
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Prefix-free complexity K is defined in a similar way, the difference being that 
the universal machine is required to be prefix-free. 

In this paper, the constant hidden in the 0{ ) notation only depends on the 
universal Turing machine. 

In this paper, by convention logO = 0. 

2.2 Symmetry of Information and the Chain Rule 

All the forms of the Symmetry of Information that we are aware of have been 
derived from the chain rule and this paper is no exception. The chain rule states 
that C(xy \ n x ,n y ) w C(x \ n x ) + C(y \ x,n y ). For us it is of interest to 
have an accurate estimation of the relation. It is immediate to see that 
C(xy | n X: n y ) < C(y \ n y ) + C(x \ y, n x ) + 2C^(y \ n y ) + 0(1). If we show a 
form of the converse inequality 

C(xy | n x ,n y ) > C(x \ n x ) + C(y \ x,n y ) — (small term), (5) 

then we deduce that C{x \ n x ) — C(x | y,n x ) < C(y \ n y ) — C(y \ x,n y ) + 
2C ( - 2 \y | n y ) + (small term), i.e., 

I(y : x) < I(x : y) + 2C (2) (y | n y ) + (small term). 

Thus our efforts will be directed to establishing forms of the relation <j5j) in which 
(small term) is indeed small. 

3 Proof of the Main Theorem 

We demonstrate TheoremQ] using the standard proof method of the Kolmogorov- 
Levin Theorem. 

Let t x — C(x | n x ),t y = C(y \ x,n y ) and t = C(xy \ n x ,n y ). We take 
w = t x +t y — t, which is the term called (small term) in equation ([5]), and plays 
the role of randomness deficiency. Our goal is to show that 

w = 0{\ogI(x : y)) + 0(log \n x - n y \) + 5(x,y), 

from which the Main Theorem follows (using the discussion in Section |2T2|) . 

We assume that w > 0, otherwise there is nothing to prove. We will need 
to handle information (\n x — n y \,t x ,t y ,w,b)), where & is a bit that indicates 
if n x > n y or not. Let A be a string encoding in a self-delimiting way this 
information, and let A be the length of A. Note that 

A < 21ogu; + 0(C (2) ( X I Ti x )+C^(y\n y )+\ogI(x:y)+\og\ 

T^x fly | ) • 

We build a 2™ x x 2" M boolean table, with rows and columns indexed by the 
strings in {0, 1}™ X and, respectively, {0, and we set the value of cell (u,v) 
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to be 1 if C(uv \ n x ,n y ) < t, and to be otherwise. Let S be the set of 1-cells, 
and S u be the set of 1-cells in row u. Note that 

\S\ < 2 t+1 . 

Let m be defined as 2 m_1 < \S X \ < 2™. We take F to be the set of "fat" rows, 
i.e., the set of rows having more than 2 m_1 1-cells. We have \F\ < \S\/2 m ~ 1 < 
2*— m+2_ that x is in F, and that the elements of F can be effectively 

enumerated given the information A and m. It follows that, given A and m, the 
string x can be described by its index in the enumeration of F. This index can 
be written on exactly t — m + 2 bits, so that knowing t which can be deduced 
from A, we can reconstruct m. It follows that 

C(x | n x ,A) < t-m + 2 + 0(l). (6) 

Next we note that y is in Sx and the elements of S x can be enumerated given x 
and A. It follows that y can be described by its index in the enumeration of S x . 
We obtain 

C(y | x,A) < m + 0(l). (7) 
From Equations ((6|) and (0, we conclude that 

C{x\n x ,A) + C(y\x, A) < t + 0(1) = t x + t y - w + 0(1). 

Note that t x = C(x | n x ) < C(x \ n x ,A) + O(A) and t y = C(y \ x,n y ) < C(y 
x, A) + O(A). We obtain t x + t y < t x + t y — w + 0(A). Taking into account the 
estimation for A, we have w — O(logiu) = 0(C^(x \ n x ) + C^(y \ n y )+\ogI(x : 

y) + log 

Tl X fly |), from which, w = 0{C^ 2 \x \ n x ) + C^ 2 \y \ n y ) + \ogI(x : 
y) + log \ n x — n y \), as desired. 

4 A van Lambalgen Theorem for prefix-free complexity 

In this section we prove Theorem [3l 

We introduce first the following notation: d is a self delimited encoding of 
the integer d. We can have \d\ < 21og<i + 4. 

For a string u, let A u (d) = {v G {0, 1}™ | K (uv \ n) < In — d}, and let 
F(d) = {ue{0,l} n \ \A u (d)\ >2 n - d }. 

Consider the following program p\ 1 which has x of length n on the oracle 
tape (as conditional information): 

On input the pair of positive integers (d, i), encoded as d bin(i), check if i 
is written on n — d bits. If not, diverge. If yes, enumerate strings u of length n 
such that K(xu \ n) < 2n — 2d. Output the i-th string in the enumeration. 

Note that the domain of pi is prefix-free. Let c\ be the length of program pi. 

Let pi be the following program, which has n on the oracle tape (as condi- 
tional information): 
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On input the pair of positive integers (d, i), encoded as d bin(i), check if i is 
written on n — d bits. If not, diverge. If yes, enumerate the elements of F(d) and 
output the i-ih one. 

Program pi is prefix-free. Let c 2 be its length. 

Let d = max{2(c + 4 + Ci), 2(c + 4 + c 2 )}. 

If there is some i such that p\(do,i) outputs u, then K(u x) < n — d + 
2 log do + 4 + C\ < n — c. So, there is no i such that pi(do, i) outputs y. This can 
happen only if either 

(a) K(xy n) > 2n — 2do, or 

(b) K(xy | n) < 2n — 2do but there are 2 n ~ d ° strings u which are enumerated 
before y. 

If (a), we are done. 

We show that (b) is impossible. 

Indeed, if (b) holds, then 1^(^)1 > 2™"* and thus x is in F(d ). Also 
\F(d )\ < 2 2n - 2do /2"- do = 2 n - d °. It follows that for some i, program p 2 on 
input (d , i) outputs x. But then K(x \ n) < n — d + 2 \ogd + 4 + c 2 < n — c, 
which is a contradiction. 

5 Final remarks 

For symmetry of information, the argument from |Zimllj based on randomness 
extractors, a concept introduced and studied intensively in computational com- 
plexity, can be replaced by the simpler and more direct argument suggested by 
Sasha Shen and presented here. Nevertheless, we still think that extractors play 
a role in Kolmogorov complexity theory. They have been used in Kolmogorov 
complexity extraction (see [FHP+lllHPVlllMusl2IZiml0dlZim09j . and the sur- 
vey paper [ZimlOaj ) and, of course, this is not surprising given the analogy be- 
tween Kolmogorov complexity extraction and randomness extraction. But they 
have also been used for obtaining results that are not immediately connected 
to randomness extraction, such as counting results regarding dependent and 
independent strings jZimlObj . and independence amplification [ZimlOcj . 
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